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Four factors in teaching intern effectiveness, as measured by a Praxis Ill-similar instrument, were found among observa¬ 
tional data of teaching interns during the 2010 spring semester. Those factors were lesson planning, teacher/student re¬ 
flection, fairness & safe environment, and professionalism/efficacy. This factor analysis was as much of a statement about 
effective teaching as it is about the technical aspects of an instrument utilized to assess it. Forty-one percent of effective 
teaching was found to be in the lesson planning. 

Keywords: effective teaching, supervision of interns, efficacy, safe school environment, teacher reflection, higher order 
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A t our university, we are constantly looking for ways to help teacher education candidates improve their teach¬ 
ing. As is probably the case in most teacher education units in the United States, our College of Education uses 
an observation form for assessing teacher intern performance and for giving feedback. When the Formative 
Observation and Intervention form was created several years ago, it was constructed so that items and domains had a 
great resemblance to the Pathwise evaluation (ETS, 1996). Accordingly, out of respect for intellectual property rights, 
we obtained written permission from the Educational Testing Service before beginning to use it with our candidates. 
This form has become useful not only for assessing intern performance, but also for identifying the most salient ele¬ 
ments of effective teaching. In other words, the form identifies what is really being identified as effective in teaching. 

Pathwise was developed through Educational Testing Service as an observation system to gather rich, research- 
based, objective classroom data based on evidence stemming from the effective teaching research (Chan, 1998). 
The effectiveness of teachers during classroom settings is rated as a category one, category two, or category three, 
depending upon very specific scoring criteria (ETS, 1996), with a category one denoting an unacceptable level of 
effectiveness. The assessment of teaching competency is thus a very authentic portrayal of teaching performance, 
since a minimum of subjectivity is employed. In addition to the 19 heavily research-based items related to the Path- 
wise system, two items were added locally for administrative and pragmatic reasons: one under Domain A, to denote 
total preparedness to teach, and another under Domain D, to denote the candidate's consistency in meeting profes¬ 
sional responsibilities. 

The observation form was used to collect data on 21 research-based items of teacher performance. These 21 areas 
were grouped into four domains: (A) Organizing Content for Student Learning; (B) Creating an Environment for Stu¬ 
dent Learning; (C) Teaching for Student Learning; and (D) Teacher Professionalism. The items (not yet the factors) of 
the observation form are shown in Table 1. Since the data obtained using the Formative Observation and Intervention 
form were used to make personnel decisions about candidates, we decided to study it in depth, using candidate data 
from the Spring Semester of 2010. We felt that, by doing this study, we could gain insight into the characteristics of 
effective teaching in addition to exploring some technical aspects of the instrument. 

A PRIORI ASSUMPTIONS 

Factor analysis can be used to test whether initial assumptions about a factor structure of an assessment instrument 
have empirical validity. Our assumptions were as follows: 

1. Four factors would be found, corresponding to the four domains of Pathwise. 
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2. The items that measured these factors would be located within the domain structure suggested by Pathwise. 

3. The two items that had been added locally would not "load" (correlate) significantly upon the rest of the fac¬ 
tor structure. 
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4. Decisions about the factor structure would not be based heavily upon the two locally-developed items alone. 

DEFINITIONS 

Domain: A collection of five or more items on the Formative Observation and Intervention form designed to assess the 
same construct. The number of items on the form exceeded the minimum number of three items to create a compo¬ 
nent (expected factor), as described by Hatcher and Stepanski (p. 460). 

Effective teaching: An assessment of teaching using the Formative Observation and Intervention form (sometimes 
referred to simply as the form) which yielded measurements of 2 or above in every one of 21 items on the form. 
Teaching was not regarded as effective if there was not enough evidence during an observation to support a cat¬ 
egory of at least a 2 in each and every one of the 21 items. 

Factor: A mathematical communality with an Eigenvalue of at least 1. On th e Formative Observation and Intervention 
form, a mathematical communality that accounted for at least 1/21 st of the variance of the entire 21 -item instrument 
used to measure teacher effectiveness. 

Factor name: The name given to a collection of items from the form identified during the factor analysis process 
whose items have a statistically significant (p<.01, n=130, one-tailed test) (Ferguson, p. 494) correlation to the factor 
and which seem to best typify the construct of the five items most correlated to the factor. 
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PURPOSE OF THE STUDY 

The principal purpose of our study was to determine if there were factor loadings on this measure of effective teach¬ 
ing and, if principal factors were found, to determine what those factors were by carefully assigning names to them. 

METHOD 

Hatchett and Stepanski (1994, p. 461) state that for factor analysis, the sample size should be the larger of 100 subjects 
or five times the number of variables being analyzed. Five times 21 items is 105. There were 130 teaching interns in 
the sample, and 416 teaching observations recorded, so the sample was more than adequate in size to accommodate 
this type of analysis. Methodologically this study should be considered a "common factor analysis" (Ingram, 2011). 
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Participants 

Participants were 63 early childhood, 9 middle level, and 58 secondary education interns, a total of 130 senior intern 
candidates. They were assigned to school campuses in the Western part of Arkansas, particularly along the 1-40 cor¬ 
ridor from Morrilton westward to the Arkansas-Oklahoma state line. All were assigned to accredited public schools 
and in content areas appropriate to their majors and expected licensures. Placement was done through the office of 
Teacher Education Student Services at the university. All public school and university faculty who participated in any 
direct way in intern evaluations were made thoroughly familiar with the Pathwise Evaluation System from the Edu¬ 
cational Testing Service through professional development experiences provided through the College of Education. 
The items of the form and their organization into subscales called domains are shown in Table 1. 

Materials and Procedures 

Before interns located to their respective placements, they were briefed about the expectations for the field experi¬ 
ence. Early childhood majors and middle level majors enrolled in a 16-week course for 15 and 12 semester hours, re¬ 
spectively; secondary majors enrolled in a nine-semester hour course encompassing a 12-week internship. Second¬ 
ary majors completed an on-campus course in public school law, history and philosophy of education, and content 
area reading before beginning their 12-week internship. All interns had completed substantially all of the require¬ 
ments for their respective majors except for the internship itself. 
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Table 1 

Item Specification and Split-half Reliability for a Performance-based Assessment of Teacher Effectivenss 


Item 


Subscale: Domain A, Organizing Content For Student Learning 

A1. Demonstrates knowledge of students' backgrounds, awareness of diversity in planning 
lessons 

A2. Prepare clear learning objectives appropriate for all students 
AB. Connect past, present, future content 

A4. Vary methods and materials for learning... developmental^ appropriate 
A5. Align learning goals with assessments... systematic, monitoring, diagnostic 
A6. Total preparedness for teaching 

Subscale: Domain B. Creating Environment for Student Learning 

B1. Models and promotes fairness with and among all students 

B2. Generates a working rapport with all students 

B3. Establishes high realistic expectations for all students 

B4. Exercises consistent, appropriate behavior management 

B5. Construct safe environment beneficial to learning for all students 

Subscale: Domain C: Teaching for Student Learning 

Cl. Clear Goals & Instructional Procedures 

C2. Makes content Comprehensible, Meaningful Engagements, Connections 
C3. Encourage all students to Extend thinking, Questioning, Critical thinking, Creative 
thinking 

C4. Monitor understanding, give specific Feedback, and Adjust for all students 
C5. Use instructional time effectively, Effective pacing, Time on Task 

Subscale: Domain D: Professionalism 

D1. Reflect on extent of goals met 

D2. Initiates modifications, accepts responsibility, efficacy 

D3. Build professional relationships, collaborates 

D4. Parent/guardian communication 

D5. On time, professional appearance, meets deadlines, follows policies 


Odds-Evens correlation _ 0.967, N=416obs. _ 

Note. Categories for each item were 1 insufficiently motivated and insufficiently knowledgeable to perform in 
classrooms unless assisted. 2=sufficiently motivated and knowledgeable to perform and performs appropriately 
in most classroom situation, meeting most learners' needs. 3=very well motivated, very knowledgeable about 
performance, and performs capably and flexibly in varied classroom situations with all learners. 


The Formative Observation and Intervention form was used by campus-based and field-based supervisors for evalu¬ 
ation purposes and to provide feedback to interns. For the purposes of this study, we decided to use the form to 
investigate the factor structure of effective teaching, using data from 130 interns of the spring semester of 2010. It 
was the intent of the supervisory experience to observe each intern at least four times while the intern was teaching; 
this occurred in most but not entirely all instances. Prior to this investigation, a previous study utilizing the same data 
had been done to determine the reliability, validity, and suitability of the Formative Observation and Intervention form 
in our application of it. These facets of the form were believed to be more than adequate (Womack, Flanna, Woodall, 
& Callaway, 2011). 

Artifact Reliability. The uncorrected split-half reliability of the Formative Observation and Intervention form was 
0.976 with 416 usable observations. The standard error of measurement was 2.6 points out of 63 possible points on 
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the entire 21 -item instrument. It appeared that the assessment was reliable. 

Artifact validity. All items on the form were mapped to the state's licensing standards and to the Praxis III (Pathwise) 
assessments. These mappings were recorded on several documents that became part of the teacher education unit's 
electronic exhibits pursuant to accreditation by the National Council for Accreditation for Teacher Education (NCATE) 
and by the State. 


RESULTS 

Data from 416 observations of 130 candidates were obtained during the spring semester of 2010. These occurred as 
faculty or clinical practice instructors completed four cycles of evaluations while observing interns in teaching situ¬ 
ations. 

Factor Loading 

The principal purpose of our study was to determine if there were factor loadings on this measure of effective teach¬ 
ing and, if principal factors were found, to determine what those factors were by carefully assigning names to them. 
Procedure FACTOR of the Statistical Analysis System was used to discover factors, using the suggested prior commu- 
nality estimate of one and a minimum Eigenvalue of one (Hatcher & Stepanskie, 1994). 

Table 2 


Eigenvalues of the Correlation Matrix (N=416 observations from 130 teaching interns) 


Factor 

Eigenvalue 

Difference 

Variance 
accounted for 

Cumulative % 

1 

8.63922 

7.28065 

41.14% 

41.14 

2 

1.35857 

0.09839 

6.47 

47.61 

3 

1.26018 

0.21271 

6.00 

53.61 

4 

1.04748 

0.06676 

4.99 

58.60 

5 

0.98072 

0.03118 

4.67 

63.27 


SAS output indicated that were likely four factors within the observational data from the interns that met these crite¬ 
ria. The fifth factor was indicated on Table 2 to show the reader where the Eigen break was. The scree plot was some¬ 
what consonant with that finding while indicating the presence of an initial large factor (Figure 1) that accounted for 
41% of the variance in teaching effectiveness scores. 

Factor Detection 

As Hatcher and Stepanski (1994) and Ingram (2011) indicate, interpretation of factors and of items correlating with 
factors is subjective. This may seem counter to the appearance of the mathematical precision of the output of a pro¬ 
gram like PROC FACTOR, but researchers, given a few suggestions from the statistical literature, are left to adopt their 
own criteria for factors and items. We determined that we would recognize a factor if it had an Eigenvalue of at least 
one, appeared distinct on the scree plot, and accounted for at least 5% of the variance. The scree plot (Figure 1) is a 
depiction of the variance extracted at each stage of the factor analysis. Hatcher and Stepanski say "The word 'scree' 
refers to the loose rubble that lies at the base of a cliff. When performing a scree test, you normally hope that the 
scree plot will take the form of a cliff: At the top will be the eigenvalues for the few meaningful components, followed 
by a break (the edge of the cliff). The bottom of the cliff will be like the scree: eigenvalues for the trivial components" 
(p. 473). We determined that we would recognize an item as being associated with a factor if its correlation with a 
factor reached statistical significance at the .01 level. We planned to name a factor in special consideration of its five 
greatest correlates (assuming there would be at least five), in view of the a priori domains from which the items came, 
and in view of the language of the items. The number five was chosen because of the original minimum of five items 
per domain on the form. 
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Figure 1. Scree Plot of Eigenvalues for Ratings of Teacher Intern Performance 


Initial Factor Method: Principal Components 
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Item-Factor Identification 

Items from the Formative Observation and Intervention form were allowed to remain in the factor structure if they cor¬ 
related significantly (critical r=.230, p<.001) with the factor. Statistical significance is not mandated in factor-naming, 
but it is a standard that is commonly used. Fifteen items correlated significantly with the first factor (see Table 3), a 
factor that accounted for 41 % of the total variance. 

Factor Naming 

The first factor was named "lesson planning." In referencing the correlations to the items on the observation form, 
two of the top five correlations were with items that dealt very obviously with planning (A2, A5). B3 (challenging 
learning expectations) usually occur as a result of careful lesson planning. D2 and D4 could be considered extensions 
of planning in that planning promotes efficacy and a sense of capability in reaching out to parents. 
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Table 3 

Rotated Factor Pattern 
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Factor 

1 

Planning 

2 

Reflection 

3 

Fair/Safe 

4 

Professionalism 

Items/correlations 

A3/0.67 

C3/.77 

B1/.77 

D5/.71 


C4/.66 

D1/.66 

B4/.63 

D3/.64 


A4/.62 

D2/.66 

B5/.54 

D2/.46 


D2/.59 

C5/.51 

A2/.46 

A1/.45 


C1/.57 

D4/.51 

D3/.44 

D4/.42 


C2/.56 

C1/.45 

A5/.39 

A4/.35 


A6/.53 

C2/.44 

B2/.36 

B3/.33 


B2/.51 

B4/.43 

A1/.35 

B2/.29 


B5/.50 

A6/.39 

D2/.39 

A2/.28 


C5/.50 

A5/.35 

A3/.29 

A6/.27 


A5/.47 

C4/.25 

D1/.27 

A3/.27 


A2/.47 

* 

C5/.24 

D1/.27 


B3/.46 

* 

* 

* 


D4/.45 

* 

* 

* 


A1/.43 

* 

* 

* 


* 

* 

* 

* 


* correlation was not significant 


Using the rotated varimax factor pattern, the other three factors were also named. The number of items correlating sig¬ 
nificantly with the remaining three factors was considerably less. The second factor correlated significantly with items 
C3, D1, D2, C5, and D4. The second factor was named "Teacher and student reflection" in consideration of the language 
of the items about teacher reflection on goals met, initiation of modifications for students' needs, and student higher 
order thinking. The third factor was named "Fairness/safe environment" in view of the language of most of the items 
contributing to its variance. The third factor correlated significantly with items B1, B4, B5, A2, and D3. Three of these 
are addressed in Domain B, Creating Environment for Student Learning. Domain B was measured during the dynamics 
of actual instructional events. The fourth factor correlated with items D5, D3, D2, A1, and D4. The fourth factor was 
named "Professionalism and efficacy" in deference to the predominant language of the items most associated with 
it—"on time, professional appearance, follows policies... builds professional relationships, collaborates ... accepts 
responsibility, efficacy... reflects on goals met." 

In an effort to better visualize which items actually loaded with which factors, we constructed a simple incidence table 
(Table 4). It seemed apparent that items correlated with factors were spread across rather than within what had been 
considered a priori to be in different domains. We were able to see some interrelationships that make up the complex 
task called "teaching." Planning, for instance, touches not only items A1 through A6, but also on the rapport that teach- 
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ers are able to build with students (B2), the framing of challenging learning expectations (B3), planning for physical 


safety (B5), the making of content comprehensible (C2), and five other items. 

Table 4 

Factors Loaded on by Each Item 


Item 

Factor 1 

Factor 2 

Factor 3 

Factor 4 


Planning 

Reflection 

Fair/Safe 

Professionalism 

A1. Awareness of student diversity 

X 


X 

X 

A2. Prepare clear learning objectives 

X 


X 

X 

A3. Connect past, present, future content 

X 


X 

X 

A4. Vary methods/ materials for learning 

X 

X 


X 

A5. Align learning goals with assessments 

X 

X 


X 

A6. Total preparedness for teaching 

X 

X 


X 

B1. Models and promotes fairness 



X 


B2. Rapport with all students 

X 


X 

X 

B3. Challenging learning expectations 

X 

X 


X 

B4. Consistent behavior management 


X 

X 


B5. Physical environment, safety 

X 


X 


Cl. Clear goals & instructional procedures 

X 

X 



C2. Makes content comprehensible 

X 

X 



C3. Critical thinking, creative thinking 


X 



C4. Teachable moments, monitor & adjust 

X 

X 



C5. Effective pacing, time on task 

X 

X 

X 


D1. Reflect on extent of goals met 


X 

X 

X 

D2. Accepts responsibility, efficacy 

X 

X 

X 

X 

D3. Professional relationships, 
collaborates 

X 


X 

X 

D4. Parent/guardian communication 

X 

X 


X 

D5. On time, prof, appearance, policies 
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DISCUSSION 

Given the nature of the assessment instrument—one designed to assess effective teaching, with its reliability and 
validity, this study was not only a study on technical issues, but also on the nature of effective teaching itself. As 
mentioned earlier, there were four a priori assumptions about the factor structure of the instrument that were tested 
in this factor analysis: 
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1. Four factors would be found, corresponding to the four domains of Pathwise. 

2. The items that measured these factors would be located within the domain structure suggested by Pathwise. 

3. The two items that had been added locally would not load significantly upon the rest of the factor structure. 

4. Decisions about the factor structure would not be based heavily upon the two locally-developed items alone. 

With regard to assumption one, four factors were found, but they did not nearly correspond to the subscales sug¬ 
gested by our Praxis Ill-like instrument, the Formative Observation and Intervention form. Regarding assumption two, 
the items that loaded most heavily on each of the four factors were not all from the respective domains of the form; 
rather they were scattered across several domains. The first and largest factor, that of planning, had item loadings 
from all four domains. Lesson planning correlated significantly with 15 of 21 items of our research-based instrument 
that were designed to assess effective teaching. Only in the fourth factor were most of the five most-correlated items 
from the domains that had been suggested a priori. Regarding the third assumption about the two locally developed 
items—ones that had not been expected to load or correlate with the rest of the instrument—item A6 as a reflection 
of total preparedness to teach a specific lesson was at least significantly correlated to three of the four factors. The 
locally-added item on Domain D, item D5 about being on time and meeting professional responsibilities, loaded on 
and was significantly correlated on factor four, being the most correlated of the items within the factor. Thus, speak¬ 
ing to the fourth assumption, while decisions about the factor structure ended up being related to the two locally-de¬ 
veloped items, the data did not suggest that these two items were "out of place," com pared to the 19 ETS-based items. 

The Value of Lesson Planning 

Forty-one percent (41%) of the variance in effective teaching in our interns was accounted for by lesson planning. 
That is, before they walked into a classroom and uttered the first word of the day, 41 % of student learning has already 
been decided by the preparedness or lack thereof of the teacher for that specific moment. Intuitively we in teacher 
education have emphasized to novice teachers the importance of careful and thorough lesson planning. With the 
findings of this study, that importance need no longer be one advanced only by intuition. Lesson planning as a sig¬ 
nificant endeavor goes beyond just deciding which method or which activity to utilize in a lesson. There was little 
evidence in our findings to promote any particular methodology as a panacea for teaching any or all subjects. Rather, 
lesson planning touches the eventual method of assessment that students will face, planning for safety in the physi¬ 
cal environment, planning for fairness, planning for challenging learning expectations and for higher-order thinking, 
planning for effective pacing and time on task, and more. Teachers who are constantly prepared for the next day, 
week, and month of teaching find it easy to approach and interact with parents. It is easier to cultivate rapport with 
students when "What will I be doing next period?" is not a real concern. For these and other reasons, the value of les¬ 
son planning can hardly be overstated. 

The Value of Reflection and Higher-order Thinking 

Teacher reflection and student higher order thinking, the second largest factor, accounted for 6.47% of the total 
variance in teacher effectiveness. Reflection enables teachers at all experience levels to gain much more from their 
experiences than just the initial exposure. Our interns are required to write reflections about the events of each day. 
The value of higher order thinking for both the teachers and their students is well established in the literature. 

The Value of Fairness and of a Safe-School Environment 

Fairness and safe-school environment accounted for 6% of the variance in teacher effectiveness. Students need to be 
treated fairly by teachers and by other students. Students need to be assured that their work will be evaluated fairly 
by teachers. They also need to be assured that they will not be bullied by classmates. Most states have passed laws 
during the past decade to deal with bullying. Teachers and administrators should do their part in enforcing these 
long-overdue laws. 

Professionalism, Responsibility, and Efficacy 

Professionalism accounted for about 5% of the variance in teacher effectiveness. At least two Domain D items loaded 
on each of the four factors. Professionalism must be part of everything that a teacher does. Professionalism is ex- 
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pressed in the effort level that teachers show in always being prepared for classes, in the preparedness that teachers 
show in adopting and implementing classroom management strategies, in the ways that teachers treat other teach¬ 
ers and administrators, and in the ways that teachers seek interactions with parents. 

Other Variance 

About 41 % of the variance was not accounted for by the model. This variance in teaching effectiveness was scattered 
among many small categories. With the high reliability and small error of measurement, it was not believed that 
measurement error was a large factor. Many small but essential behaviors comprise effective teaching. They add in 
small but incremental ways to the total amount of student learning that takes place. 

CONCLUSIONS 

After years of utilizing the Formative Observation and Intervention form, this study helps the observer to be able to 
have a discussion with interns about the importance of planning. As professionals, we often try to stress this to pre¬ 
service teachers, but without much success. Now we have a number that we can place on what is really important 
and to what degree planning is important. That number is 41% of their success. This information can help to give 
concrete evidence to students as well as teachers how important their planning can be. 

Knowing what areas make a real difference can also help with planning on the part of the university. It seems crucial 
to spend time training our pre-service teachers in the skill of planning. Therefore, it is important to spend the time in 
our courses with specific training on the importance and the methodology in specifically how to plan for teaching. 

The values of teacher reflection and of student higher-order thinking are well established in the literature. When 
teachers reflect, they are able to "re-experience" a lesson many times over and to learn from both their successes and 
failures. Students absorb, rearrange content, and store it in long-term memory in ways that are personal and idio¬ 
syncratic to each of them. Reflection and higher-order thinking should continue to be emphasized, regardless of the 
grade level of the teachers and students involved. 

Fairness and safe-school environment have arisen as significant factors especially in the past twenty years. Incidents 
such as those in Jonesboro, Arkansas, Columbine, Colorado, and Virginia Tech have given a heightened awareness of 
the need to feel secure. Without these feelings of security, higher-order thinking and reflection are not likely to occur 
(Maslow, in Ormrod, 2004, pp. 432-433). In the past generation, our society has become more aware of bullying and 
the long-term, negative effects of bullying. Students need to feel safe not only from the forces outside of the class¬ 
room, but also from those that are within. 

The factor of professionalism, responsibility, and efficacy lies not only in the interactions with other teachers and 
parents, but goes much deeper. It involves the teacher caring about their profession. It involves the teacher taking 
on the responsibility for their students learning. Interestingly, without good daily planning, it is nearly impossible for 
any of this to happen. In conclusion, it all comes back to planning. Without substantial effort and skill in this area, 
the intern or teacher cannot effectively establish a classroom of learning that is fair, safe, elicits higher order think¬ 
ing, or enables students to grow in a productive manner. We, as teacher educators, must be prepared to model and 
teach these skills to our pre-service teachers and to our interns in order to enhance their opportunities to succeed 
and become effective teachers. 


RECOMMENDATIONS 

A recommendation for future research would be to further explore which kinds of planning seem to enhance teacher 
effectiveness the most. It is likely that all forms of planning are not equally productive. 
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